Priority (v1)

• Update pyproject.toml with ProbLog dependency.
• Refactor build as code run_check method into smaller subchecks.
• Use ProbLog predicates to enable prolog string to access subcheck results.
• Write ProbLog rules to evaluate the check's overall confidence score.
• Update BuildAsCodeTable to receive the information.
• Compare overall confidence score with threshold to make a decision about check passing.
• Update check_result to store confidence score.
• Intermediate problog queries that allow a decision to be made on which deployment method to store data from.
• Store results from each ci_service and compare confidence_scores to produce final result. Currently just using the first CI service.
• Handle ProbLog errors properly.
• Move subchecks to be external to BAC check.
• Add additional subchecks.
• Fetch project name from setup.py file. Timestamp check requires the project name. This is straightforward to fetch for projects where it's specified in the pyproject.toml, but less so for those that use setup.py. I developed an initial implementation for setup.py but later removed it because I realised it was only applicable for very specified scenarios. The other option was to execute setup.py to fetch the name, but for obvious reasons we shouldn't do that.

Subchecks

• Timestamp of pypi package vs pipeline run - just need to finish with the github_actions function to fetch the 5 most recent runs of a workflow file.
• Workflow trigger
• Test deployment
• Secrets

Note: for many of the sub-checks we've had to do a check for deploy action method vs deploy command as there can be multiple methods used in a workflow. Need to figure out a more streamlined way to handle this.

Later

• Update expected outcomes for integration tests.
• Address the assumption that a project must have a deployment method found - this limited the inference. However, the challenge associated with this is that most sub-tasks rely on the existence of a particular workflow or workflow step to check. Perhaps could store information per workflow? and build up evidence in this way.
• We added tox -e release as a deploy command, however release is just a target pointing to a section in a tox.ini file, so potentially need to investigate parsing these files so verify that it contains deploy commands.
• Most of the functionality of Poetry overlaps with pip anyway, and there's little differentiation between the two in terms of project setup - i.e. Poetry projects can still use the same external GHAs etc. to deploy. Could be potential to have generic python class and use the inference to instead make a decision about which is used but not until later in the check.

Notes:

• Branch multiple-deploy-commands-collected-test stores some changes from the proof of concept for performing inference of multiple deployment commands (that was used in the Micronaut case study).